Formal testing of software is necessary not only to ensure the quality of a software product but to verify that the various components or modules of the software meet their design requirements and integrate together correctly to produce the desired effect. Formal testing may include the design, development, and execution of a variety of test cases. Each test case generally targets a specific element of functionality or individual interface of a software module, and includes a number of test conditions, a test script to be executed to test the conditions, and the expected results for each major step in the script. The test script can be executed manually or by automated testing tools. The use of automated testing, while often requiring more upfront effort, provides for more complete testing of software modules as well as more efficient regression testing of the software as changes to the modules are implemented. In addition, automated testing allows for testing metrics to be automatically collected for each execution of a test script, such as the date when executed, the execution time, and the result (pass or fail).
The various test scripts utilized in automated testing may further be collected into test suites. For example, one test suite may be created to include all test scripts for a specific software module, and another created for a particular element of functionality across all modules. Test suites may also be created to include test scripts based on the time of execution and the functionality tested. For example, one test suite may include test scripts that test the critical functions of a software product in the least amount of time, while another includes test scripts that test a broader range of functions but require more time to complete. Accordingly, the various test suites will be executed with different levels of frequency, depending upon the amount of time available for testing and the required scope of coverage.
Some test scripts may be prone to producing “false failures.” False failures are failures not due to a product bug but due to external factors. For example, the execution of a test script may result in a failure due to locks placed on resources by other test scripts being executed simultaneously, timing issues in user interface elements due to the speed of the automated testing software, or unavailability of external resources, such as a networked resource. Other test scripts may not be effective at identifying product bugs because they are subject to random failures or intermittent race conditions. In addition, as individual software modules and components are changed over time, individual test scripts may be rendered less effective, for example, by requiring an inordinate amount of time to test an element of functionality or interface that is no longer critical to the software product. Because of the number of scripts that may be utilized in the testing of a software product, it can be difficult to identify the test scripts that are prone to producing false failures or those that have become less effective in identifying product bugs.
It is with respect to these considerations and others that the disclosure made herein is presented.